-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Caikit Standalone ServingRuntime #343
Support for Caikit Standalone ServingRuntime #343
Conversation
- Added Caikit Standalone ServingRuntime as a model option - Outlined endpoints available for use - Provided example output for endpoints - Added metrics information
…l-serving-platform.adoc
…l-serving-platform.adoc
…l-serving-platform.adoc
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Show resolved
Hide resolved
…l-serving-platform.adoc
modules/viewing-metrics-for-the-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/viewing-metrics-for-the-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/viewing-metrics-for-the-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/viewing-metrics-for-the-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/viewing-metrics-for-the-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/viewing-metrics-for-the-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
modules/accessing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
Outdated
Show resolved
Hide resolved
* `:443 caikit.runtime.Nlp.NlpService/RerankTasksPredict` | ||
+ | ||
ifdef::upstream[] | ||
NOTE: By default, the Caikit Standalone ServingRuntime exposes REST endpoints for use. To use gRPC protocol, manually deploy a custom Caikit Standalone ServingRuntime. For more information, see link:{odhdocshome}/serving-models/#adding-a-custom-model-serving-runtime-for-the-single-model-serving-platform_serving-large-models[Adding a custom model-serving runtime for the single-model serving platform]. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably settle on 'HTTP endpoints', rather than 'REST endpoints'. I think HTTP will be the protocol that's indicated next to the inference endpoint in the dashboard.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, thanks for confirming, @christinaexyou 👍 In that case, let's do the opposite and title the bullet list of endpoints to match? And the example queries, too?
Description
Update documentation to indicate that we now support the Caikit Standalone ServingRuntime.
How Has This Been Tested?
Merge criteria: